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1. SUMMARY REPORT TO THE MINISTER 





1.1. Background 


In January 1997, the Ministry entered into a Common Purpose Procurement (CPP) agreement 
with Andersen Consulting to implement the Business Transformation Project. This contract is 
one of the first undertaken by any government ministry in Ontario to use CPP principles. There 
have been concerns raised by the Provincial Auditor in his 1998 annual report about the 
management of the contract. 


Under CPP, a private sector vendor is selected to work closely with the Ministry to identify, 
design, develop and implement new ways of delivering services and in so doing share the 
investment in and risks and rewards of the project. The main purpose of the BTP project is to 
develop new business practices and technology to transform the Family Benefits and General 
Welfare Assistance programs into the Ontario Works Program and the Ontario Disability Support 
Program. The contract with Andersen Consulting was based on the CPP guidelines issued by the 
Management Board of Cabinet in 1995. 


1.2. Purpose 

The purpose of this review is to: 

a) Provide advice on how to ensure and measure the value of this service to the taxpayer; 

b) Review the Auditor’s findings and provide advice on additional action needed; 

c) Assess the Ministry’s strategies to manage the CPP arrangement with Andersen Consulting 
and provide advice on additional actions needed to address any gaps. 

1.3. Scope 

The focus of the review will be on the Ministry’s Business Transformation Project and the 


Ministry’s management and monitoring of the contract between the Ministry and Andersen 
Consulting. 


1.4. Objectives 


The review will determine whether: 
e The Ministry has identified the risks and appropriate controls for Contract Management, 


e The responsibility for managing the contract is well defined and the accountabilities are 
clearly identified; 
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e The financial controls, quality controls, reporting and monitoring processes are working 
effectively and efficiently; 


e The management monitoring systems have established accountability and value for money to 
ensure the best use of taxpayer dollars; 


e New steps are required regarding contract management in light of the new MBS CPP 
Guideline. 


1.5. Approach to the Review 


The review team will: 
e Interview past and current management and staff associated with the project; 
e Review ministry documents and reports; 


e Conduct an analysis of the financial and statistical data collected and assess the quality and 
effectiveness of the project so far; 


e Assess whether or not the Ministry has in place the necessary systems to ensure the integrity 
of the information being reported to senior management; 


e Ensure the methodology for determining if benefit attribution is appropriate. 


1.6. Getting it Right 


In the course of the review, we met a large and diverse group of MCSS and Andersen Consulting 
officials, a few of whom are no longer involved in the BTP. Most of those we met are diligently 
engaged in the project. Without exception, we found concerned and cooperative people focussed 
on “getting it right” as the project seeks to achieve its promise. The review concluded with the 
following key observations: 


1.6.1. BTP Return on Investment 


Our third party assessment of the BTP business case produced an expected return on investment 
of 222%. High returns of this magnitude indicate that modernizing MCSS business processes is 
long overdue. 


1.6.2. Economic Value of BTP Benefits 


Our third party assessment of the BTP business case also showed that with the realization of the 
expected cost, schedule and performance results over the project’s economic life, the BTP 
investment program should yield $346.7 million in benefits including both economic ($297.2 
million) and social ($49.5 million) benefits net of all costs (present value in 1996). 
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1.6.3. Lower Average Hourly Billing Rates 


The current average hourly billing rate for Andersen Consulting’s professional services is 
expected to fall by more than 25% by year-end 1999 based on assumptions regarding the mix of 
partner, manager, consultant and analyst personnel that Andersen plans to deploy and the 
anticipated division of labour between MCSS and Andersen personnel going forward. 


1.6.4. Improving Project Governance and Control 


The Ministry is taking several initiatives to strengthen and improve the governance and control of 
the BTP. For example, measures to apply strong functional control in finance, human resource 
management, contract administration, operations and communications are being implemented 
along with regular and rigorous project schedule, budget and resource allocation reporting to the 
Ministry management committee. 


1.6.5. Improving an Innovative Agreement 


The agreement between MCSS and Andersen Consulting was an innovative attempt at 
formulating a performance-based public/private business arrangement that, in terms of work 
product and project management methodology, has produced good quality results. At the time of 
its formulation, MCSS’ expected outcomes or business results were not known and were not 
specified in the agreement. The new Common Purpose Procurement (CPP) guidelines (October 
1998) require such specification in CPP arrangements and, with the recent validation of the BTP 
business case, they are now known and can be included. 


1.6.6. Effectiveness Measures and Reporting 


The Ministry has confirmed that the large benefits of the BTP arise in the form of (i) productivity 
gains in the MCSS administration of OW and ODSP, and (ii) reductions in the volume of 
inaccurate payments under these programs. The Ministry management has adopted specific 
effectiveness measures and intends to report effectiveness within the Canadian Comprehensive 
Auditing Foundation’s guidelines. 


1.6.7. Readiness and Schedule Risk 


There is acknowledgement generally that MCSS was not adequately prepared to perform its 
expected role at the outset of the BTP. This led to significant delay in the course of the first two 
years of the project. While some slippage may be expected in long-term projects like BTP, the 
dominant risk going forward is that of slippage in the roll-out and ramp-up of new business 
processes and support technology. While aggressive risk management and mitigation of 
schedule risk itself carries a cost, each month of accelerated roll-out and ramp-up yields five per 
cent greater net benefit to the Ontario taxpayer. 


1.6.8. Conclusion 


“Getting it Right” necessitates assiduous MCSS management attention to getting qualified 
people deployed to the BTP, to getting project decisions made, to getting the corporate staff 
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functions fully engaged and to enabling the necessary intergovernmental arrangements — all ina 
timely way. The will to do so appears to be strong. 


1.7. Summary of Review Recommendations 


Following are our recommendations, fifteen in all, organized in order of the three purposes of the 
review — that is, those related to ensuring and measuring the value of the service; those related to 
additional actions needed further to the Provincial Auditor’s findings; and, those related to the 
management of the CPP arrangement with Andersen Consulting. 


In those cases where MCSS management has taken the initiative to implement a recommendation 
and where that implementation is substantially completed, the recommendation is marked with a 
double asterisk (**). In those cases where implementation has commenced but is not yet 
completed, the recommendation is marked with a single asterisk (*). At this critical juncture and 
with the Service Delivery Model (SDM) design in hand: 










Review Recommendation 1 — MCSS - Andersen Agreement — Ensuring Value 






In light of past experience, the parties should, in considering improvements to their agreement 
that better ensure the value of the service, take account of the following recommended 
parameters: 


The results of the revalidated BTP business case including the effectiveness measures 
(business results) expected therein; 


The schedule risks (including those related to municipal readiness and access to third party 
data) exposed by the risk analysis conducted as part of the business case revalidation; 







Within reasonable limits, the government, represented by the Minister, would undertake to 
account for the cost and benefit effects of any proposed legislative or regulatory changes on 
the financial outcomes of the BTP before proceeding; 













Contemporaneously with the availability of statistical proof that improved effectiveness has 
arisen from the Consolidated Verification Process (CVP), verifiable statistical analysis also 
should determine the appropriate benefit duration period and actual dollar savings achieved 
by case type; 


The relatively high probability that the project could require about another three years to fully 
implement; 






The need to reflect the outcome of Review Recommendation 8 —‘Value Index” Policy — so 
that the relative values of the contributions of each party to the achievement of BTP business 
results are recognized more appropriately in benefit sharing and so that a balance between 
investment and return to each party is maintained throughout the project; 
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Upon mutual agreement to the scope of the SDM following completion of its design, the 
parties firmly commit to its full implementation. Such implementation is expected to be 
achieved within the current benefits cap assuming no further schedule delays; 






A lead role for MCSS managers and project staff in the SDM roll-out phase specifically and 
in change management programming generally; 







Prior to proceeding with the build and roll-out phases of the SDM implementation, a 
readiness assessment of MCSS’ capacity to perform the roles contemplated for it in the 
project plan is conducted and MCSS’ readiness confirmed; 











Delays in decision-making or the provision to the project of the necessary qualified resources 
in a timely manner are costly and such costs ought to be borne by the offending party . In this 
regard, if necessary, consideration should be given to the validity of the benefits cap and of 
the 2-year limit on the benefits earning period following the term of the agreement as 
effective means of recovering benefits lost due to project delays caused by either party. The 
benefits earning period could be lengthened or shortened or the benefits cap raised or 
lowered, if necessary, based on delays caused by MCSS or Andersen respectively; 













The estimated six month unproductive delay in the course of the first two years of the BTP 
attributable to the lack of readiness of MCSS to perform its role should be recognized as a 
cost to Andersen in the context of these parameters; 






Performance standards for communications (status reporting, issue resolution, etc.) and 
decision-making (resource deployment, work product approvals, task order approvals, 
acceptance, etc.) should be more aggressive (timing, frequency, escalation, etc.) than 
currently formulated in the agreement; and 











After due notice is taken of any other amendments that may be made based on these and, 
possibly, other parameters, should agreement to the scope of the SDM and to its full 
implementation within the current benefits cap not be possible, the agreement should be 
terminated and competitive proposals sought for the implementation of the SDM design and 
scope acceptable to MCSS; 








The following set of recommendations (2 to 12 inclusive) arise from our review of the Provincial 
Auditor’s findings and represent advice on additional action needed. 
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Review Recommendation 2 — Business Case Development and Risk Analysis ** 





Business case analysis ought to be performed by MCSS to identify and select those investments 
most beneficial to the achievement of its defined desired business results (outcomes). The 
Ministry should adopt a relevant means by which risks can be identified and managed for 
investment projects like BTP both at the point of the initial investment decision and at every 
incidence of significant change during project execution. Such a means is the performance of 
risk analysis as part of business case development. Risk analysis is central to business case 
development and any risk-sharing business arrangements. Risk analysis is a particularly valid 
means by which to account quantitatively and transparently for uncertainty and risk in complex 
projects like BTP. 










With risk analysis, it is reasonable to expect that, retrospectively, the risks and financial effects of 
the provision that enables Andersen Consulting to increase its billing rates from time to time to 
the level of its standard “published” rates would have been transparent to MCSS officials and, 
possibly, mitigated before the agreement was executed. 






Review Recommendation 3 — BTP Business Case Re-validation ** 


The Ministry ought to apply its Business Case Development and Risk Analysis policy to the re- 
validation of the BTP business case and, in so doing, achieve the following objectives: 


1. Provide the quantitative framework within which architecture, technology, business process 
and implementation options and choices can be made with due regard for value-for-money 


and effectiveness. 


Provide the framework within which quantitative aims for BTP effectiveness can be 
established as management decisions. 


Provide the framework within which quantitative aims for return on investment (ROI) and 
risk can be established as management decisions. 


Provide the framework within which progress toward effectiveness aims can be monitored. 


Provide the quantitative framework within which returns on investment can be measured and 
monitored. 


Provide the operational business framework within which the MCSS Senior Financial Officer 
(SFO) can monitor financial outcomes. 


Provide the operational effectiveness framework upon which the auditor can rely for 
transparency and effectiveness measurement. 


Provide the framework within which the social benefits of BTP can be measured and 
accounted for in business and policy decisions. 
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Review Recommendation 3 —(cont'd) 


9. To quantify and monitor both MCSS and Andersen Consulting ROI. 


And, as part of the annual report to the Management Board of Cabinet under the new Common 
Purpose Procurement Guidelines (October 1998), seek explicit approval of the re-validated BTP 
business case. 





Review Recommendation 4 — Document Management * 










A review of the effectiveness of the BTP Document Management Process should be conducted 
by the Director, Information Systems Branch early in 1999 with a report of findings and 
recommendations to the Project Leader - BTP, Project Director - BTP and the Assistant Deputy 
Minister, Business Planning and Corporate Services. 


Review Recommendation 5 — MCSS’ Future Common Purpose Procurements 


In its application to future Common Purpose Procurements, MCSS’ Business Case Development 
and Risk Management Policy (see Review Recommendation 2) will mean that: 


e The Ministry’s business case would be disclosed in a discussion paper and form part of the 
vendor consultation suggested in Management Board’s revised CPP guidelines; 


The Ministry’s business case (modified, where applicable, to reflect the vendor consultation) 
would be included in the subsequent request for proposals; and 


Vendor recommendations that would improve on the Ministry’s business case would be 
evaluated as part of the determination of the risk/reward implications and overall value to the 
Ministry. 


Review Recommendation 6 — BTP Business Case Custodian * 


The Ministry’s Senior Financial Officer (SFO) should be the custodian of the BTP business case. 
The SFO should provide the necessary business case and risk analysis services when Ministry 
management decision support is sought. 
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Review Recommendation 7 — Maximizing Return on Investment 





In its performance of the business case analysis in response to Review Recommendation 3, the 
Ministry ought to include all project costs. And, in assessing its risks associated with computer 
hardware, third party software, production support, help desk services, annual application 
maintenance costs and any applicable taxes, the Ministry should consider its options to minimize 
cost exposure and maximize the return on its investments. 







Review Recommendation 8 — “Value Index” Policy * 


The Ministry, in comparing the value of its contribution to a BTP-like project with that of 
consultants when both parties are contributing to project costs in anticipation of proportionate 
benefits-sharing, should develop and apply a “value index” for its personnel reflecting their 
relative value to the project. Should the current agreement be amended, it should reflect such a 
“value index” with its concomitant effect on benefits-sharing. 









Review Recommendation 9 -- Financial Functional Control * 





The Ministry’s Senior Financial Officer (SFO) should exercise functional control with respect to 
the determination, qualification and allocation of BTP costs and benefits. In so doing, the SFO 
should be satisfied that all Ministry units and personnel expending effort or money in support of 
BTP see to it that all qualified expenditures are recorded and charged to BTP. 


Review Recommendation 10 — Additional Work ** 


In every case where additional work is incorporated in the BTP, consonant with Review 
Recommendation 2, the benefits and costs of that work ought to be subjected to complete 
business case and risk analysis in the context of the current project business case (the base case). 
In so doing, the available feasible alternatives should be analyzed to determine which yields the 
most attractive Net Present Value (NPV) or net benefit to the Ministry or to the government as a 
whole. 


Review Recommendation 11 — Effective Program Delivery * 


The Ministry management team ought to significantly strengthen the governance and control of 
BTP fully expecting that the specific measures set out in Review Recommendation 14 will hasten 
the efficient delivery of needed support to the Ontario Works and Ontario Disability Support 
Programs. 
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The next set of recommendations (12 to 15 inclusive) arise from an assessment of the Ministry’s 
Strategies to manage the CPP arrangement with Andersen Consulting and represent advice on 
additional actions needed to address any gaps. 


Review Recommendation 12 - BTP Effectiveness Reporting ** 


The management of the Ministry ought to be ready to make representations on the effectiveness 
of BTP throughout its development and operational life. It is recommended that MCSS adopt the 
following Canadian Comprehensive Auditing Foundation — suggested effectiveness attributes for 
BIR: 


Management Direction: the extent to which the objectives (outcomes) of BTP are clear, well- 
integrated and understood, and appropriately reflected in MCSS’ plans, structures, delegations of 
authority and decision-making processes; 


Relevance: the extent to which BTP continues to make sense in regard to the problems and 
conditions to which it is intended to respond; 


Appropriateness: the extent to which the design of BTP, and the level of effort being made, are 
logical in light of the specific objectives to be achieved; 


Achievement of Intended Results: the extent to which BTP objectives have been or are 
expected to be realized; 


Acceptance: the extent to which the stakeholders in BTP judge it to be satisfactory (to their 
needs); 


Secondary Impacts: the extent to which other significant consequences, either intended or 
unintended, positive or negative, have arisen as a result of BTP; 


Costs and Productivity: the relationship among costs, inputs and outputs resulting from BTP; 


Responsiveness: BTP’s ability to adapt to changes in such factors as case volumes, program 
design, available funding, technology, etc.; 


Financial Results: the matching of, and accounting for costs and benefits arising from BTP and 
the accounting for and valuation of assets and the return to the Crown on BTP investments; 


Working Environment: the extent to which the BTP provides a productive work atmosphere for 
MCSS employees, appropriate opportunities for their development and success, and promotes 
commitment and initiative; 
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Review Recommendation 12 — (cont'd) 


Protection of Assets: the extent to which important assets — such as intellectual property, sources 
of supply, key personnel, important records and information, software, etc. are safeguarded so 
that BTP is protected from the danger of losses that could threaten its success, credibility, 
continuity, etc.; 


Monitoring and Reporting: the extent to which key measures pertaining to performance and 
organizational strength are identified, reported and monitored; 


To portray BTP’s effectiveness in a balanced way, MCSS management should adopt all the 


above attributes so that third parties, especially legislators, can form a balanced judgement on 
effectiveness. 


Review Recommendation 13 — BTP Governance and Control* 


The work of BTP ought to be integrated within existing MCSS processes and structures. Such 


integration means that, for example: 


The ADM, Business Planning and Corporate Services will exercise full functional control 
and provide project support services in financial, human resource management and contract 
management functions; 


The ADM, Program Management will exercise full functional control and provide project 
support in operations during the design, build and, especially, the roll-out phases of the BTP; 
and 


The Director, Communications and Marketing will exercise full functional control and 
provide project support services in communications planning and delivery. 


In addition, the recommended governance of the BTP includes nine core elements: the Ministry 
Management Committee, the Project Sponsor, the BTP Executive Committee, the Project 
Leader, the Project Executive Director; the Project Director; the Quality Council; the Contract 
and Risk Manager, and, the Internal Audit Mandate. The roles of each of these core elements are 
set out in Sections 3.4.1 to 3.4.9 inclusive. 





Review Recommendation 14 — Relationship Management * 












Within the governance structure recommended for BTP, a key to the effectiveness of the 
relationship between MCSS and Andersen Consulting is the quality of communications between 
the two parties and the decision-making that arises therefrom. Just as the project plan sets 
rigorous performance standards for the work itself, so should the parties set performance 
standards for communications (status reporting, issue resolution, etc.) and decision-making 
(resource deployment, work product approvals, task order approvals, acceptance, etc. ) more 
aggressive (timing, frequency, escalation, etc.) than those found in the current Agreement. 
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Review Recommendation 15 -- Automating Social Assistance Project (ASAP) * 


As the project enters its third year, the scope and significance of BTP is much better known and 
the need for focussed management to achieve its outcomes is palpable. Similarly, ASAP’s needs 
are significant and worthy of focussed management attention in their own right. However, ASAP 
is dealing with the “now” needs of MCSS while BTP is the “future” of MCSS program delivery. 
Accordingly, it is recommend that: 


ASAP should report to the Assistant Deputy Minister, Business Planning and Corporate 
Services; and 


BTP should continue to report to the Assistant Deputy Minister Social Assistance and 
Employment Opportunities as is now the case. 


These two executives should, by formal agreement, ensure that current and future 
interdependencies are identified and attended to in a timely and responsive manner. 





1.8. Methodology for Determining Benefit Attribution - Summary 


In an effort to ensure the methodology for determining if benefit attribution is appropriate, we 
analyzed the BTP methodology used in attributing benefits to Consolidated Verification Process 
(CVP) and Change Reporting in terms of the significance of the incremental amount of 
terminations claimed as crystallized benefits and compared it with our test of statistical 
Significance and robustness. The analysis sought to determine the accuracy, consistency and 
Statistical validity of data and focussed on the statistical significance of the claimed crystallized 
terminations. 


Multivariate statistical analysis indicates that improved effectiveness can be confirmed in relation 
to the introduction of Change Reporting 1 and Change Reporting 2 in September 1997 and that 
Significant provable business and social benefits are accruing accordingly. On the other hand, 
statistical analysis does not yet confirm that improved effectiveness has arisen from CVP, as 
introduced in April 1998. Since a full year or so of experience may reasonably be required in 
order to ascertain such proof, the absence of confirmed CVP benefits as of November 1998 is 
notable though not alarming. 


The full analysis of the methodology for determining benefit attribution is found in Volume 3, 
Appendix C. 
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